Scaling Up Sign Spotting Through Sign Language Dictionaries

Abstract The focus of this work is sign spotting –given a video an isolated sign, our task to identify whether and where it has been signed in continuous, co-articulated language video. To achieve task, we train model using multiple types available supervision by: (1) watching existing footage which sparsely labelled mouthing cues; (2) reading associated subtitles (readily translations the content) provide additional weak-supervision ; (3) looking up words (for no examples are available) visual dictionaries enable novel spotting. These three tasks integrated into unified learning framework principles Noise Contrastive Estimation Multiple Instance Learning. We validate effectiveness approach on low-shot benchmarks. In addition, contribute machine-readable British Sign Language (BSL) dictionary dataset signs, BslDict , facilitate study task. dataset, models code at project page.